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SYSTEMS AND METHODS FOR SURGICAL NAVIGATION 



BACKGROUND 

In recent years, the medical community has been increasingly focused on 
5 minimizing the invasiveness of surgical procedures. Advances in imaging technology and 
. instrumentation have enabled procedures using minimally-invasive surgery with very 
small incisions. Growth in this category is being driven by a reduction in morbidity 
relative to traditional open procedures, because the smaller incisions minimize damage to 
healthy tissue, reduce patient pain, and speed patient recovery. The introduction of 
10 miniature CCD cameras and their associated micro-electronics has broadened the 
application of endoscopy from an occasional biopsy to full minimally-invasive surgical 

ablation and aspiration. 

Minimally-invasive endoscopic surgery offers advantages of a reduced likelihood 
of intraoperative and post-operative complications, less pain, and faster patient recovery. 

15 However, the small field of view, the lack of orientation cues, and the presence of blood 
and obscuring tissues combine to make video endoscopic procedures in general 
disorienting and challenging to perform. Modem volumetric surgical navigation 
techniques have promised better exposure and orientation for minimally-invasive 
procedures, but the effective use of current surgical navigation techniques for soft tissue 

20 endoscopy is still hampered by two difficulties: accurately tracking all six degrees of 
freedom (DOF) on a flexible endoscope within the body, and compensating for tissue 
deformations and target movements during an interventional procedure. 
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To illustrate, when using an endoscope, the surgeon's vision is limited to the 
camera's narrow Held of view and the lens is often obstructed by blood or fog, resulting 
in the surgeon suffering a loss of orientation. Moreover, endoscopes can display only 
visible surfaces and it is therefore often difficult to visualize tumors, vessels, and other 
5 anatomical structures that lie beneath opaque tissue (e.g., targeting of pancreatic 
adenocarcinomas via gastro-intestinal endoscopy, or targeting of submucosal lesions to 
sample peri-intestinal structures such as masses in the liver, or targeting of subluminal 

lesion in the bronchi). 

Recently, image-guided therapy (IGT) systems have been introduced. These 

10 systems complement conventional endoscopy and have been used predominantly in 
neurological, sinus, and spinal surgery, where bony or marker-based registration can 
provide adequate target accuracy using pre-operative images (typically 1-3 mm). While 
IGT enhances the surgeon's ability to direct instruments and target specific anatomical 
structures, in soft tissue these systems lack sufficient targeting accuracy due to intra- 

15 operative tissue movement and deformation. In addition, since an endoscope provides a 
video representation of a 3D environment, it is difficult to correlate the conventional, 
purely 2D IGT images with the endoscope video. . Correlation of information obtained 
from intra-operative 3D ultrasonic imaging with video endoscopy can significantly 
improve the accuracy of localization and targeting in minimally-invasive IGT procedures. 

20 Until the mid 1990's, the most common use of image guidance was for 

stereotactic biopsies, in which a surgical trajectory device and a frame of reference were 
used. Traditional frame-based methods of stereotaxis defined the intracranial anatomy 
with reference to a set of fiducial markers, which were attached to a frame that was 



2 



screwed into the patient's skull. These fiducials were measured on pre-operative 
tomographic (MRI or CT) images. 

A trajectory-enforcement device was placed on top of the frame of reference and 
used to guide the biopsy tool to the target lesion, based on prior calculations obtained 
5 from pre-operative data. The use of a mechanical frame allowed for high localization 
. accuracy, but caused patient discomfort, limited surgical flexibility, and did not allow the 
surgeon to visualize the approach of the biopsy tool to the lesion. There has been a 
gradual emergence of image guided techniques that eliminate the need for the frame 
altogether. The first frameless stereotactic system used an articulated robotic arm to 
10 register pre-operative imaging with the patient's anatomy in the operating room. This was 
followed by the use of acoustic devices for tracking instruments in the operating 
environment. The acoustic devices eventually were superceded by optical tracking 
systems, which use a camera and infrared diodes (or reflectors) attached to a moving 
object to accurately track its position and orientation. These systems use markers placed 
15 externally on the patient to register pre-operative imaging with the patient's anatomy in 
the operating room. Such intra-operative navigation techniques use pre-operative CT or 
MR images to provide localized information during surgery. In addition, all systems 
enhance intra-operative localization by providing feedback regarding the location of the 
surgical instruments with respect to 2D preoperative data. 
20 Until recently, volumetric surgical navigation has been limited by the lack of the 

computational power required to produce real-time 3D images. Today the use of various 
volumetric imaging modalities has progressed to permit the physician to visualize and 
quantify the extent of disease in 3D in order to plan and execute treatment. Today, 



3 



systems are able to provide real-time fusion of pre-operative 3D data with intraoperative 
2D data images from video cameras, ultrasound probes, surgical microscopes, and 
endoscopes. These systems have been used predominantly in neurological, sinus, and 
spinal surgery, where direct access to the pre-operative data plays a major role in the 
5 execution of the surgical task. This is despite the fact that, because of movement and 
. deformation of the tissue during the surgery, these IGT procedures tend to lose their 
spatial registration with respect to the pre-operatively acquired image. 
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SUMMARY 

In one aspect, a method for assisting a user in guiding a medical instrument to a 
subsurface target site in a patient includes generating one or more intraoperative images 
on which a spatial feature of a patient target site can be indicated, indicating a spatial 

5 feature of the target site on said image(s), using the the spatial feature of the target site 
. indicated on said image(s) to determine 3-D coordinates of the target site spatial feature 
in a reference coordinate system, tracking the position of the instrument in the reference 
coordinate system, projecting onto a display device, a view field as seen from a known 
position and, optionally, a known orientation, with respect to the tool, in the reference 

10 coordinate system, and projecting onto the displayed view field, indicia whose states are 
related to the indicated spatial feature of the target site with respect to said known 
position and, optionally, said known orientation, whereby the user, by observing the 
states of said indicia, can guide the instrument toward the target site by moving the 
instrument so that said indicia are placed or held in a given state in the displayed field of 

15 view. 

The generating includes using an ultrasonic source to generate an ultrasonic 
image of the patient, and the 3-D coordinates of a spatial feature indicated on said image 
are determined from the 2-D coordinates of the spatial feature on the image and the 
position of the ultrasonic source. The medical instrument can be an endoscope and the 
20 view field proj ected onto the display device can be the image seen by the endoscope. The 
view field projected onto the display device can be that seen from the tip-end position 
and orientation of the medical instrument having a defined field of view. The view field 
projected onto the display device can be that seen from a position along the axis of 
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instrument that is different from the target than the tip-end position of the medical 
instrument. The target site spatial feature indicated can be a volume or area, and said 
indicia are arranged in a geometric pattern which defines the boundary of the indicated 
spatial feature. The target site spatial feature indicated can be a volume, area or point, 
5 and said indicia are arranged in a geometric pattern that indicates the position of a point 
within the target site. The spacing between or among indicia can be indicative of the 
distance of the instrument from the target-site position. The size or shape of the 
individual indicia can indicate the distance of the instrument from the target-site position. 
The size or shape of individual indicia can also be indicative of the orientation of said 

10 tool. The indicating includes indicating on each image, a second spatial feature which, 
together with the first-indicated spatial feature, defines a surgical trajectory on the 
displayed image. The instrument can indicate on a patient surface region, an entry point 
that defines, with said indicated spatial feature, a surgical trajectory on the displayed 
image. The surgical trajectory on the displayed image can be indicated by two sets of 

15 indicia, one set corresponding to the first-indicated spatial feature and the second, by the 
second spatial feature or entry point indicated. The surgical trajectory on the displayed 
image can be indicated by a geometric object defined, at its end regions, by the first- 
indicated spatial feature and the second spatial feature or entry point indicated. 

In another aspect, a system for guiding a medical instrument to a target site in a 

20 patient includes an imaging device for generating one or more intraoperative images, on 
which spatial features of a patient target site can be defined in a 3-dimensional coordinate 
system, a tracking system for tracking the position and optionally, the orientation of the 
medical instrument and imaging device in a reference coordinate system, an indicator by 
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which a user can indicate a spatial feature of a target site on such image(s), a display 
device, an electronic computer operably connected to said tracking system, display 
device, and indicator, and computer-readable code which is operable, when used to 
control the operation of the computer, to perform (i) recording target-site spatial 
5 information indicated by the user on said images), through the use of said indicator, (ii) 
. determining from the spatial feature of the target site indicated on said image(s), 3-D 
coordinates of the target-site spatial feature in a reference coordinate system, (iii) 
tracking the position of the instrument in the reference coordinate system, (iv) projecting 
onto a display device, a view field as seen from a known position and, optionally, a 
10 known orientation, with respect to the tool, in the reference coordinate system, and (v) 
projecting onto the displayed view field, indicia whose states indicate the indicated 
spatial feature of the target site with respect to said known position and, optionally, said 
known orientation, whereby the user, by observing the states of said indicia, can guide 
the instrument toward the target site by moving the instrument so that said indicia are 
1 5 placed or held in a given state in the displayed field of view. 

Implementations of the above aspect may include one or more of the following. 
The imaging device can be an ultrasonic imaging device capable of generating digitized 
. images of the patient target site from any position, respectively, and said tracking device 
is operable to record the positions of the imaging device at said two positions. The 
20 medical instrument can be an endoscope and the view field projected onto the display 
device is the image seen by the endoscope. 

In yet another aspect, machine readable code in a system designed to assist a user 
in guiding a medical instrument to a target site in a patient, said system including (a) an 
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imaging device for generating one or more intraoperative images, on which a patient 
target site can be defined in a 3-dimensional coordinate system, (b) a tracking system for 
tracking the position and optionally, the orientation of the medical instrument and 
imaging device in a reference coordinate system, (c) an indicator by which a user can 
indicate a spatial feature of a target site on such image(s), (d) a display device, and (e) an 

. electronic computer operably connected to said tracking system, display device, and 
indicator, and said code being operable, when used to control the operation of said 
computer, to (i) record target-site spatial information indicated by the user on said 
image(s), through the use of said indicator, (ii) determine from the spatial feature of the 
target site indicated on said image(s), 3-D coordinates of the target-site spatial feature in 
a reference coordinate system, (iii) track the position of the instrument in the reference 
coordinate system, (iv) project onto a display device, a view field as seen from a known 
position and, optionally, a known orientation, with respect to the tool, in the reference 
coordinate system, and (v) project onto the displayed view field, indicia whose states 
indicate the indicated spatial feature of the target site with respect to said known position 
and, optionally, said known orientation, whereby the user, by observing the states of said 
indicia, can guide the instrument toward the target site by moving the instrument so that 
said indicia are placed or held in a given state in the displayed field of view. 

In yet another aspect, a method for assisting a user in guiding a medical 
instrument to a subsurface target site in a patient includes indicating a spatial feature of a 
patient target site on an intraoperative image, determining 3-D coordinates of the patient 
target site spatial feature in a reference coordinate system using the spatial feature of the 
target site indicated on the intraoperative image, determining a position of the instrument 
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in the reference coordinate system, projecting onto a display device a view field from a 
predetermined position relative to the instrument in the reference coordinate system, and 
projecting onto the view field an indicia of the spatial feature of the target site 
corresponding to the predetermined position. 

5 

Advantages of the system may include one or more of the following. The system 
enhances intra-operative orientation and exposure in endoscopy, in this way increasing 

10 surgical precision and speeding convalescence, which will in turn reduce overall costs. 
The ultrasound-enhanced endoscopy (USEE) improves localization of targets, such as 
peri-lumenal lesions, that lie hidden beyond endoscopic views. The system dynamically 
superimposes directional and targeting information, . calculated from intra-operative 
ultrasonic images, on a single endoscopic view. Magnetic tracking and 3D ultrasound 

15 technologies are used in conjunction with dynamic 3D/video calibration and registration 
algorithms for precise endoscopic targeting. With USEE, clinicians use the same tools 
and basic procedures as for current endoscopic operations, but with a higher probability, 
of accurate biopsy, and an increased chance for the complete resection of the 
abnormality. The system allows accurate soft-tissue navigation. The system also 

20 provides effective calibration and correlation of intra-operative volumetric imaging data 
with video endoscopy images. 

Other advantages may include one or more of the following. The system acquires 
external 3D ultrasound images and process them for navigation in near real-time. The 
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system allows dynamic target identification on any reformatted 3D ultrasound cross- 
sectional plane. The system can automatically track the movement of the target as tissue 
moves or deforms during the procedure. It can dynamically map the target location onto 
the endoscopic view in form of a direction vector and display quantifiable data such as 
distance to target. Optionally, the system can provide targeting information on the 
dynamic 3D ultrasound view. The system can virtually visualize the position and 
orientation of tracked surgical tools in the ultrasound view, and optionally also in the 
endoscopic view. It can overlay dynamic Doppler ultrasound data, rendered using 
intensity based opacity filters, on the endoscopic view. 
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DESCRIPTION 

Fig. 1 shows an exemplary process 5 to guide a medical instrument to a desired 
position in a patient. First, one or more intraoperative images of the target site are 
acquired (10). Next, the process registers the intraoperative images, the patient target 
5 site, and the surgical instruments into a common coordinate system (20). The patient, 
. the imaging source(s) responsible for the intraoperative images and surgical tool must all 
be placed in the same frame of reference (in registration), and this can be done by one of 
a variety of methods, among them: 

1. Use a wall-mounted tracking device for tracking patient, imaging source(s), 
10 and the surgical tool, e.g., endoscope. 

2. Track only the position of the tool, and place the tool in registration with the 
patient and imaging source by touching the tool point to fiducials on the body and 
to the positions of the imaging source(s). Thereafter, if the patient moves, the 
device could be registered" by tool-to-patient contacts. That is, once the images 

15 are made, from known coordinates, it is no longer necessary to further track the 

position of the image source(s). 

3. The patient and image sources are placed in registration by fiducials on the 
patient and in the images, or alternatively, by placing the imaging device at 
known coordinates with respect to the patient. The patient and tool are placed in 

20 registration by detecting the positions of fiducials with respect to the tool, e.g., by 

using a detector on the tool for detecting the positions of the patient fiducials. 
Alternatively, the patient and an endoscope tool can be placed in registration by 
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imaging the fiducials in the endoscope, and matching the imaged positions with 
the position of the endoscope. 

Referring back to Fig. 1, the process then tracks the position of the surgical 
instrument with respect to the patient target site (30). A magnetic tracking system is used " 
5 to track the endoscope for navigation integration in one implementation. The system 
- provides a magnetic transducer into the working channel at the endoscope tip, positioning 
the field generator so that the optimal sensing volume encompasses the range of sensor 
positions. In one implementation that provides for six degrees of freedom (6 DOF), a 
miniaturized magnetic tracking system with metal insensitivity known as miniBIRD 500 
10 (available from Ascension Technology Corp. of Burlington, VT) can be used. The 
tracking system may be calibrated using a calibration jig. A calibration target is modified 
from a uniform to a non-uniform grid of points by reverse-mapping the perspective 
transform, so that the calibration target point density is approximately equal throughout 
the endoscope image. The calibration jig is waterproofed and designed to operate in a 
1 5 submerged environment. Where appropriate, calibration will be performed while the jig 
is immersed in a liquid with refractive properties similar to the operating environment. 

In one embodiment, an ultrasound calibration system can be used for accurate 
reconstruction of volumetric ultrasound data. An optical tracking system is used to 
measure the position and orientation of a tracking device that will be attached to the 
20 ultrasound probe. A spatial calibration of intrinsic and extrinsic parameters of the 
ultrasound probe is performed. These parameters are used to transform the ultrasound 
image into the co-ordinate frame of the endoscope's field of view. In another 
embodiment, a magnetic tracking system is used for the ultrasound probe. Using only one 
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tracking system for both the endoscope and the ultrasound probe reduces obstructions in 
the environment, and avoids a line-of-sight tracking requirement. In another 
embodiment, tracking of the probe is done using an optical tracking system. The 
calibration of the 3D probe is done in a manner similar to a 2D ultrasound probe 
calibration using intensity-based registration. Intensity-based registration is fully 
- automatic and does not require segmentation or feature identification. In the typical 2D 
case, acquired images are subject to scaling in the video generation and capture process. 
This transformation and the known position of the phantom's tracking device are used to 
determine the relationship between the ultrasound imaging volume and the ultrasound 
probe's tracking device. Successful calibration requires an unchanged geometry. The 
phantom will be designed to withstand relocation and handling without deformation. A 
quick-release clamp attached to the phantom will hold the ultrasound probe during the 
calibration process. 

A spatial correlation of the endoscopic video with dynamic ultrasound images is 
then done. The processing internal to each tracking system, endoscope, and ultrasound 
machine causes a unique time delay between the realtime input and output of each 
device. The output data streams are not synchronized and are refreshed at different 
intervals. In addition, the time taken by the navigation system to acquire and process 
these outputs is stream-dependant. Consequently, motion due to breathing and other 
actions can combine with these independent latencies to cause real-time display of 
dynamic device positions different to those when the imaging is actually being acquired. 

A computer is used to perform the spatial correlation. The computer can handle a 
larger image volume, allowing for increased size of the physical imaged volume or higher 
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image resolution (up to 512 * 512 * 512 instead of 256 * 256 * 64). The computer also 
provides faster 3D reconstruction and merging, and a higher-quality perspective volume 
rendering at a higher frame rate. The computer time-stamps and buffers the tracking and 
data streams, then interpolating tracked device position and orientation to match the 
5 image data timestamps. 

In determining the required time offset, the ultrasound probe is moved across a 
step surface in the calibration phantom to create a temporal step function in both the 
tracking system and image data stream. The relative delay is determined by comparing 
the timestamps of the observed step function in each data stream. The endoscope latency 
10 is determined similarly using the same phantom. This should be performed whenever the 
ultrasound system is reconfigured. The endoscope latency will not need to be recalculated 
unless the endoscope electronics are changed, however. The patient is imaged through 
the ultrasound probe, and the endoscope becomes the frame of reference for the surgeon. 
The important information is contained in the dynamic relationship of the ultrasound data 
15 to the endoscope video, which is known through calibration and tracking of both devices. 

Turning now to Fig. 1, one or more images of the patient target site are shown on 
a display device (40). A user indicates a spatial feature of the patient target site on the 
images of the patient target site (50), and an indicia is projected on the images relating 
the position and orientation of the surgical instruments to the spatial feature of the patient 

20 target site (60). 

The method dynamically tracks and targets lesions in motion beyond the visible 
endoscopic view. When a target is identified, the subregion surrounding the target in the 
ultrasound volume will be stored as a reference, together with the tracked orientation of 
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the volume. A subregion of each successively-acquired ultrasound volume, centered at 
the target position in the preceding volume, will be re-sampled using the orientation of 
the reference target subregion. 3D cross-correlation of the re-sampled subregion with the 
reference subregion will be used to find the new location of the target. This dynamic 
5 tracking will follow each target over time; if the system is displaying target navigation 
data, the data will change in real time to follow the updated location of the target relative 
to the endoscope. 

Vascular structures return a strong, well differentiated Doppler signal. The 
dynamic ultrasound data may be rendered in real time using intensity-based opacity 

10 filters, making nonvascular structures transparent. This effectively isolates the vascular 
structure without requiring computationally-demanding deformable geometric models for 
segmentation, thus thesystem can follow movements and deformations in real time. 

The system of Fig. 1 allows a user such a surgeon to mark a selected target point 
or region on intraoperative ultrasonic images (one or more 3-D ultrasound images). The 

15 designated target point or region is then displayed to the surgeon during a surgical 
operation, to guide the position and orientation of the tool toward the target site. In a first 
general embodiment, the target area is displayed to the user by displaying a field 
representing the patient target area, and using the tracked position of the tool with respect 
to the patient to superimpose on the field, one or more indicia whose position in the 

20 displayed field is indicative of the relative position of the tool with respect to the marked 
target position. In a second general embodiment, the tool is equipped with a laser 
pointer that directs a laser beam onto the patient to indicate the position and orientation of 
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a trajectory for accessing the target region. The user can follow this trajectory by 
aligning the tool with the laser-beam. 

In the embodiment where the tool is an endoscope, the displayed image is the 
image seen by the endoscope, and the indicia are displayed on this image. The indicia 

5 may indicate target position as the center point of the indicia, e.g., arrows, and tool 
- orientation for reaching the target from that position, by the degree of elongation of 
arrows, such that the indicia are brought to equal sizes when the tool is properly oriented. 
Alternatively, the indicia may indicate the surface point for entry and the elongation of 
the arrows, the tool orientation-trajectory for reaching the target from that surface point 

10 In one embodiment that enables surgeons to visualize a field of view of the 

surgical endoscope overlaid with volumetrically-reconstructed medical images of a 
localized area of the patient's anatomy. Using this volumetric navigation system, the 
surgeon visualizes the surgical site via the surgical endoscope, while exploring the inner 
layers of the patient's anatomy through the three-dimensionally reconstructed pre- 

15 operative MRI or CT images. Given the endoscope's position and orientation, and given 
the characteristics of the camera, a perspective volume-rendered view matching that of 
the optical image obtained by the endoscope is rendered. This system allows the surgeon 
to virtually fly through and around the site of the surgery to visualize alternative 
approaches and qualitatively determine the best one. The volumetrically reconstructed 

20 images are generated using intensity based filtering and direct perspective volume 
rendering, which removes the need for conventional segmentation of high-contrast 
images. The real-time 3D-rendered radiographic reconstruction images matched with the 
intra-operative endoscopic images provide a new capability in minimally-invasive 
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endoscopic surgery. Since hitting vascular structures remains the greatest hazard in 
endoscopic procedures, this new technology represents a marked improvement over 
conventional image-guidance systems, which generally display 2D reconstructed images . 
In operation, and with respect to an embodiment using ultrasonic images, the user 

5 makes a marking on the image corresponding to the target region or site. This marking 
. may be a point, line or area. From this, and by tracking the position of the tool in the 
patient coordinate system, the system functions to provide the user with visual 
information indicating the position of the target identified from the ultrasonic image. 

The navigation system operates in three distinct modes. The first is target 

10 identification mode. The imaged ultrasound volume will be displayed to allow the 
surgeon to locate one or more target regions of interest and mark them for targeting. 
The system will show an interactive volumetric rendering as well as up to three user 
positionable orthogonal cross-sectional planes for precise 2D location of the target. 

In the second mode, the endoscope will be used to set the position and orientation 

15 of the frame of reference. Based on these parameters and using the optical characteristics 
of the endoscope, the system will overlay target navigation data on the endoscope video. 
This will allow the surgeon to target regions of interest beyond the visual range of the 
endoscope's field of view. Displayed data will include the directions of, and distances to, 
the target regions relative to the endoscope tip, as well as a potential range of error in this 

20 data. 

The third mode will be used to perform the actual interventional procedure (such 
as biopsy or ablation) once the endoscope is in the correct position. The interactive 
imaged ultrasound volume and cross-sectional planes will be displayed, with the location 
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of the endoscope and the trajectory through its tip projected onto each of the views. The 
endoscope needle itself will also be visible in the ultrasound displays. 

The system allows the interventional tool to be positioned in the center of the 
lesion without being limited to a single, fixed 2D ultrasound plane emanating from the 
5 endoscope tip. (That 2D view capability can be duplicated by optionally aligning a cross 
. sectional ultrasound plane with the endoscope.) In the first implementation of the 
endoscope tracking system, a magnetic sensor will need to be removed from the working 
channel in order to perform the biopsy, and the navigation display will use the stored 
position observed immediately prior to its removal. In another embodiment, a sensor is 
10 integrated into the needle assembly, which will be in place at calibration. 

The system provides real-time data on the position and orientation of the 
endoscope, and the ultrasound system provides the dynamic image data. The tip position 
. data is used to calculate the location of the endoscope tip in the image volume, and the 
probe orientation data will be used to determine the rendering camera position and 
15 orientation. Surgeon feedback will be used to improve and refine the navigation system. 
Procedure durations and outcomes will be compared to those of the conventional biopsy 
procedure, performed on the phantom without navigation and image-enhanced endoscopy 
assistance. 

When a target is identified, the subregion surrounding the target in the ultrasound 
20 volume will be stored as a reference, together with the tracked orientation of the volume. 
A subregion of each successively-acquired ultrasound volume, centered at the target 
position in the preceding volume, will be re-sampled using the orientation of the 
reference target subregion. 
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3D cross-correlation of the re-sampled subregion with the reference subregion 
will be used to find the new location of the target. This dynamic tracking will follow 
each target over time; if the system is displaying target navigation data, the data will 
change in real time to follow the updated location of the target relative to the endoscope. 

5 Fig. 2 shows another exemplary implementation where a process 1 00 acquires one 

- or more 2D or 3D intraoperative images of the patient target site from a given orientation 
(110). Next, the process tracks the position of a surgical instrument with respect to the 
patient target site (120). The process then registers the intraoperative images of the 
patient site, the patient target site, and the surgical instrument into a common 3D 

10 reference coordinate system (130). The image of the patient target site is rendered on a 
display device (140), and a spatial feature (shape and position)of the patient target site on 
the image is specified (150). The process then correlates the position and orientation of 
the surgical instrument with respect to the target feature (160). An indicia (3D shape or 
points and lines) is projected on the intraoperative image relating the position and 

1 5 orientation of the surgical instrument to the target spatial feature (1 70). 

Exemplary user interfaces for the systems of Figs. 1-2 are shown in Figs. 3-4. 
Fig. 3 shows an exemplary user interface (UI) for ultrasound-enhanced endoscopy. The 
left panel shows the endoscopic view with a superimposed targeting vector and a distance 
measurement. The right panels show reformatted crosssectional planes through the 

20 acquired 3D ultrasound volume. Fig. 4 shows another UI for ultrasound-enhanced 
endoscopy. The left panel shows the endoscopic view with virtual tool tracking and 
visualization and vasculature acquired through Doppler imaging. The lower right 
panel shows volume-rendered 3D ultrasound. 
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The UIs of Figs. 3-4 support interactive rendering of the ultrasound data to allow 
a user to locate and mark the desired region of interest in the ultrasound image volume. 
The UIs allow the user to locate and mark target regions of interest. Hitting vascular 
structures is a serious hazard in endoscopic procedures. Visualization of the vasculature 

5 behind the surface tissue in the endoscopic view would assist in avoiding the vascular 
% structures (anti-targeting). 

The invention has been described in terms of specific examples which are 
illustrative only and are not to be construed as limiting. The invention may be 
implemented in digital electronic circuitry or in computer hardware, firmware, software, 

10 or in combinations of them. Apparatus of the invention may be implemented in a 
computer program product tangibly embodied in a machine-readable storage device for 
execution by a computer processor; and method steps of the invention may be performed 
by a computer processor executing a program to perform functions of the invention by 
operating on input data and generating output. Suitable processors include, by way of 

15 example, both general and special purpose microprocessors. Storage devices suitable for 
tangibly embodying computer program instructions include all forms of non-volatile 
" memory including, but not limited to: semiconductor memory devices such as EPROM, 
EEPROM, and flash devices; magnetic disks (fixed, floppy, and removable); other 
magnetic media such as tape; optical media such as CD-ROM disks; and magneto-optic 

20 devices. Any of the foregoing may be supplemented by, or incorporated in, specially- 
designed application-specific integrated circuits (ASICs) or suitably programmed field 
programmable gate arrays (FPGAs). 
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From the a foregoing disclosure and certain variations and modifications already 
disclosed therein for purposes of illustration, it will be evident to one skilled in the 
relevant art that the present inventive concept can be embodied in forms different from 
those described and it will be understood that the invention is intended to extend to such 
further variations. While the preferred forms of the invention have been shown in the 
. drawings and described herein, the invention should not be construed as limited to the 
specific forms shown and described since variations of the preferred forms will be 
apparent to those skilled in the art. Thus the scope of the invention is defined by the 
following claims and their equivalents. 
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ABSTRACT 

Systems and methods are disclosed for assisting a user in guiding a medical 
instrument to a subsurface target site in a patient by indicating a spatial feature of a 
patient target site on an intraoperative image, determining 3-D coordinates of the patient 
5 target site spatial feature in a reference coordinate system using the spatial feature of the 
. target site indicated on the intraoperative image, determining a position of the instrument 
in the reference coordinate system, projecting onto a display device a view field from a 
. predetermined position relative to the instrument in the reference coordinate system, and 
projecting onto the view field an indicia of the spatial feature of the target site 
10 corresponding to the predetermined position. 
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10 -Generate intraoperative image/s of the target site. 


20- Register the intraoperative images, the patient 
target site, and the surgical instruments into a common 
coordinate system 


30- Track the position of the surgical instrument with 
respect to the patient target site 


40- Generate image/s of the patient target site on a 
display device 


50- Indicate a spatial feature of the patient target 
site on the images of the patient target site 


60- Project indicia on the images relating the position 
and orientation of the surgical instruments to the 
spatial feature of the patient target site 




FIG. 1 




110) Generate a 2D or 3D intraoperative images of the 
patient target site from a given orientation 




120) track the position of a surgical instrument with 
respect to the patient target site 




130) Register the intraoperative images of the patient 
site, the patient target site, and the surgical 
instrument into a common 3D reference coordinate system 




140) Display image of the patient target site on a 
display device 




150) Specify a spatial feature (shape and position) of 
the patient target site on the image 




160) Correlate the position and orientation of the 
surgical instrument with respect to the target feature 




170) Project indicia (3D shape or points and lines) on 
the intraoperative image relating the position and 
orientation of the surgical instrument to the target 
spatial feature 




FIG. 2 
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